Data-Driven Detection of Malicious Document PhD Thesis Proposal

Malcode hidden in otherwise normal appearing public documents provide both convenient and stealthy means for attackers to penetrate systems. By exploiting the ubiquitous and object-oriented approach of modern document applications and formats, malcode can reach third-party applications that may harbor exploitable vulnerabilities otherwise unreachable by network-level service attacks: by clicking a web link of any arbitrary website or even inserting media such as CD-ROMS and USB drives, a target system can be infected when a document is opened, effectively bypassing all the network firewalls and sensors. To protect against this stealthy threat, this thesis investigates the ability to detect embedded malcode in Word documents using two techniques: static content analysis using statistical models of typical document content and run-time dynamic tests. The first approach parses Word documents into its constituent objects and identifies the “sections” of the binary file content of those objects. The Anagram sensor introduced for network packet anomaly detection and an Entropy analysis technique are applied for this purpose. The second approach uses a series of dynamic tests on diverse platforms to open a document and execute its embedded malcode in diverse environments forcing the malcode to behave abnormally leading to its detection. The intent of such code to do harm is detected by observing unexpected changes it makes to the underlying platforms when it is able to execute. In addition to simply observing the system, a few techniques used to monitor virtual machines and detect anomaly are proposed in this document. A formal study using thousands of infected Word documents gathered from the wild has been performed, and several deficiencies of both approaches are identified, representing both challenges in addressing the problem and opportunities for follow on research. This document presents a twenty-four month research program for developing the major components of the proposed approaches, including a hybrid system that applies both techniques and for completing the experiments and evaluations.